Background Jobs

We start by loading the backgroundjobs library and defining a few trivial functions to illustrate things with.


In [1]:
from __future__ import print_function
from IPython.lib import backgroundjobs as bg

import sys
import time

def sleepfunc(interval=2, *a, **kw):
    args = dict(interval=interval,
                args=a,
                kwargs=kw)
    time.sleep(interval)
    return args

def diefunc(interval=2, *a, **kw):
    time.sleep(interval)
    raise Exception("Dead job with interval %s" % interval)

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print('In the background...', n)
        sys.stdout.flush()
    print('All done!')
    sys.stdout.flush()

Now, we can create a job manager (called simply jobs) and use it to submit new jobs.
Run the cell below and wait a few seconds for the whole thing to finish, until you see the "All done!" printout.


In [2]:
jobs = bg.BackgroundJobManager()

# Start a few jobs, the first one will have ID # 0
jobs.new(sleepfunc, 4)
jobs.new(sleepfunc, kw={'reps':2})
jobs.new('printfunc(1,3)')

# This makes a couple of jobs which will die.  Let's keep a reference to
# them for easier traceback reporting later
diejob1 = jobs.new(diefunc, 1)
diejob2 = jobs.new(diefunc, 2)


Starting job # 0 in a separate thread.
Starting job # 2 in a separate thread.
Starting job # 3 in a separate thread.
Starting job # 4 in a separate thread.
Starting job # 5 in a separate thread.

You can check the status of your jobs at any time:


In [3]:
jobs.status()


In the background... 0
Running jobs:
0 : <function sleepfunc at 0x102cc6848>
2 : <function sleepfunc at 0x102cc6848>
3 : printfunc(1,3)
5 : <function diefunc at 0x102cc68c0>

Dead jobs:
4 : <function diefunc at 0x102cc68c0>

For any completed job, you can get its result easily:


In [4]:
jobs[0].result
j0 = jobs[0]
j0.join?

You can get the traceback of any dead job. Run the line below again interactively until it prints a traceback (check the status of the job):


In [5]:
print "Status of diejob1:", diejob1.status
diejob1.traceback()  # jobs.traceback(4) would also work here, with the job number


In the background... 1
In the background... 2
All done!
  File "<ipython-input-5-a90bd59af669>", line 1
    print "Status of diejob1:", diejob1.status
                             ^
SyntaxError: invalid syntax

This will print all tracebacks for all dead jobs:


In [6]:
jobs.traceback()


Traceback for: <BackgroundJob #4: <function diefunc at 0x102cc68c0>>
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/bgranger/Documents/Computing/IPython/code/ipython/IPython/lib/backgroundjobs.pyc in call(self)
    489 
    490     def call(self):
--> 491         return self.func(*self.args, **self.kwargs)

<ipython-input-1-7391f8ae281b> in diefunc(interval, *a, **kw)
     14 def diefunc(interval=2, *a, **kw):
     15     time.sleep(interval)
---> 16     raise Exception("Dead job with interval %s" % interval)
     17 
     18 def printfunc(interval=1, reps=5):

Exception: Dead job with interval 1

Traceback for: <BackgroundJob #5: <function diefunc at 0x102cc68c0>>
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/Users/bgranger/Documents/Computing/IPython/code/ipython/IPython/lib/backgroundjobs.pyc in call(self)
    489 
    490     def call(self):
--> 491         return self.func(*self.args, **self.kwargs)

<ipython-input-1-7391f8ae281b> in diefunc(interval, *a, **kw)
     14 def diefunc(interval=2, *a, **kw):
     15     time.sleep(interval)
---> 16     raise Exception("Dead job with interval %s" % interval)
     17 
     18 def printfunc(interval=1, reps=5):

Exception: Dead job with interval 2

The job manager can be flushed of all completed jobs at any time:


In [7]:
jobs.flush()


Flushing 3 Completed jobs.
Flushing 2 Dead jobs.

After that, the status is simply empty:


In [8]:
jobs.status()

It's easy to wait on a job:


In [9]:
j = jobs.new(sleepfunc, 2)
print("Will wait for j now...")
sys.stdout.flush()
j.join()
print("Result from j:")
j.result


Starting job # 0 in a separate thread.
Will wait for j now...
Result from j:
Out[9]:
{'args': (), 'interval': 2, 'kwargs': {}}